Statistical Language Models for Intelligent XML Retrieval

The XML standards that are currently emerging have a number of characteristics that can also be found in database management systems, like schemas (DTDs and XML schema) and query languages (XPath and XQuery). Following this line of reasoning, an XML database might resemble traditional database systems. However, XML is more than a language to mark up data; it is also a language to mark up textual documents. In this chapter we specifically address XML databases for the storage of ‘document-centric’ XML (as opposed to ‘data-centric’ XML [42]). Document-centric XML is typically semi-structured, that is, it is characterised by less regular structure than data-centric XML. The documents might not strictly adhere to a DTD or schema, or possibly the DTD or schema might not have been specified at all. Furthermore, users will in general not be interested in retrieving data from document-centric XML: They will be interested in retrieving information from the database. That is, when searching for documents about “web information retrieval systems”, it is not essential that the documents of interest actually contain the words “web”, “information”, “retrieval” and “systems” (i.e., they might be called “internet search engines”). An intelligent XML retrieval system combines ‘traditional’ data retrieval (as defined by the XPath and XQuery standards) with information retrieval. Essential for information retrieval is ranking documents by their probability, or degree, of relevance to a query. On a sufficiently large data set, a query for “web information retrieval systems” will retrieve many thousands of documents that contain any, or all, of the words in the query. As users are in general not willing to examine thousands of documents, it is important that the system ranks the retrieved set of documents in such a way that the most promising documents are ranked on top, i.e. are the first to be presented to the user. Unlike the database and XML communities, which have developed some well-accepted standards, the information retrieval community does not have